Artificial neural networks are a class of electronic circuits which attempt to achieve human-like performance in the fields of speech and image recognition via dense interconnection of simple computational elements. The neural network structures and electronic circuits comprise the computational elements in the network and are each based on our present understanding of biological nervous systems. The promise of neural networks lies in their ability to provide parallel processing of information at high computational rates, far exceeding the performance of conventional von Neumann computers which perform a program of instructions sequentially.
Neural networks generally take the form of a matrix comprised of a set of horizontal lines which cross and are coupled to a set of vertical lines. The horizontal lines simulate the functions of axons in the cortex of the brain and provide the inputs to the network. The vertical lines simulate the function of dendrites. The vertical dendrite lines are terminated at summing devices which replicate the function of the soma, otherwise known as the neuron cell body. Examples of such networks can be found in U.S. Pat. Nos. 4,950,917, 4,906,865 and 4,904,881.
Within the neural network, electronic circuits are employed to model the function of a biological synapse. Collectively, these circuits provide resistive interconnections between the horizontal and vertical lines of the network. Individual synapse cells provide a weighted electrical connection between an input and summing element (i.e., a neuron body). The relative strength of the connection often changes during the training or learning process. The strength of the interconnection is often referred to as the weight of the memory.
Electronic synapse cells have been developed for storing connection weights in the form of electrical charge. Among the various approaches to building electrical synapse circuits which have been explored, the most promising designs employ floating gate devices. In a floating gate device current flow is modulated depending on the value of a stored electrical charge. Examples of semiconductor synapse cells which employ floating gate devices for storing weights are found in U.S. Pat. Nos. 4,956,564 and 4,961,002.
One of the most intriguing aspects of neural network circuits is their learning capability. During the learning process, the network is presented with a set of training data inputs. For each set of inputs an output of the network is calculated and compared with a desired output. The weights of the synaptic connections are then adjusted according to a weight-adjustment algorithm. A wide variety of learning algorithms have been developed--the most common of these being those which employ a "local rule" or "outer product" weight update rule where the change (or sign) in the weight for a particular synapse is proportional to the sign of the input and some output-related quantity. The output-related quantity is associated with the neuron to which it is connected and is referred to in this application as the quantity .OMEGA.. The interpretation of the quantity .OMEGA. depends on the particular learning algorithm employed. The local rule class of algorithms includes, for example, Hebbian, Boltzman, Mean Field, Delta Rule and Back-Propagation algorithms, among others.
One drawback with neural circuits to date has been their inability to include learning capabilities directly on chip. In other words, modern neural network circuits do not integrate the learning procedure on the same silicon chip as the network itself. In these circuits the weights are usually determined by simulation on a host computer, after which time they are down-loaded to the network. At least part of the reluctance to provide neural network implementations which allow learning to occur directly on chip has been the relative complexity of the algorithms involved, as well as the cautious approach of not committing to one particular type of algorithm. For instance, future research may reveal that certain present-day algorithms provide non-optimal learning.
As will be seen, the present invention discloses several learning circuits and a number of methods for analog, parallel implementation of a class of learning algorithms. Analog methods provide inherently higher density and computation speed than most digital implementations. Thus, the present invention provides a very dense network capable of high speed operation during the learning mode. Furthermore, by incorporating the learning on the chip itself, the weight adjustment process is enhanced by the parallelism inherent in the neural network architecture. Hence, the network of the present invention is truly adaptive in that weights can be adjusted in real-time.